{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f3396b6d",
   "metadata": {},
   "source": [
    "## 2.数据预处理\n",
    "为了能⽤深度学习来解决现实世界的问题，我们经常从预处理原始数据开始，⽽不是从那些准备好的张量格式数据开始，常用数据处理工具有numpy,pandas等\n",
    "\n",
    "### 2.2.1 读取数据集\n",
    "举⼀个例⼦，我们⾸先创建⼀个⼈⼯数据集，并存储在CSV（逗号分隔值）⽂件 ../data/house_tiny.csv中。以其他格式存储的数据也可以通过类似的⽅式进⾏处理。下⾯我们将数据集按⾏写⼊CSV⽂件中。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "68e8bc85",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.makedirs(os.path.join('..','data'),exist_ok=True)\n",
    "data_file = os.path.join('..','data','house_tiny.csv')\n",
    "with open(data_file,'w') as f:\n",
    "    f.write('NumRooms, Alley, Prices\\n') # 列名\n",
    "    f.write('NA,Pave,127500\\n') # 没行表示一个样本\n",
    "    f.write('1,Pave,124500\\n') \n",
    "    f.write('3,NA,123200\\n') \n",
    "    f.write('NA,NA,123200\\n') "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "057dfaeb",
   "metadata": {},
   "source": [
    "要从创建的CSV⽂件中加载原始数据集，我们导⼊pandas包并调⽤read_csv函数。该数据集有四⾏三列。其\n",
    "中每⾏描述了房间数量（“NumRooms”）、巷⼦类型（“Alley”）和房屋价格（“Price”）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "41f51282",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data dir ..\\data\\house_tiny.csv\n",
      "   NumRooms  Alley   Prices\n",
      "0       NaN   Pave   127500\n",
      "1       1.0   Pave   124500\n",
      "2       3.0    NaN   123200\n",
      "3       NaN    NaN   123200\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "print('data dir',data_file)\n",
    "data = pd.read_csv(data_file)\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46c13157",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " ### 2.2.2 处理缺失值\n",
    " 注意，“NaN”项代表缺失值。 为了处理缺失的数据，典型的方法包括插值法和删除法， 其中插值法用一个替代值弥补缺失值，而删除法则直接忽略缺失值。 在这里，我们将考虑插值法。\n",
    "\n",
    "通过位置索引iloc，我们将data分成inputs和outputs， 其中前者为data的前两列，而后者为data的最后一列。 对于inputs中缺少的数值，我们用同一列的均值替换“NaN”项。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   NumRooms  Alley\n",
      "0       2.0   Pave\n",
      "1       1.0   Pave\n",
      "2       3.0    NaN\n",
      "3       2.0    NaN\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\yangq\\AppData\\Local\\Temp\\ipykernel_1420\\3782217204.py:2: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError.  Select only valid columns before calling the reduction.\n",
      "  inputs = inputs.fillna(inputs.mean())\n"
     ]
    }
   ],
   "source": [
    "inputs,outputs = data.iloc[:,0:2],data.iloc[:,2]\n",
    "inputs = inputs.fillna(inputs.mean())\n",
    "print(inputs)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "对于inputs中的类别值或离散值，我们将“NaN”视为一个类别。 由于“巷子类型”（“Alley”）列只接受两种类型的类别值“Pave”和“NaN”， pandas可以自动将此列转换为两列“Alley_Pave”和“Alley_nan”。 巷子类型为“Pave”的行会将“Alley_Pave”的值设置为1，“Alley_nan”的值设置为0。 缺少巷子类型的行会将“Alley_Pave”和“Alley_nan”分别设置为0和1。"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   NumRooms   Alley_Pave   Alley_nan\n",
      "0       2.0            1           0\n",
      "1       1.0            1           0\n",
      "2       3.0            0           1\n",
      "3       2.0            0           1\n"
     ]
    }
   ],
   "source": [
    "inputs = pd.get_dummies(inputs, dummy_na=True)\n",
    "print(inputs)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 2.2.3 转换为张量格式\n",
    "现在inputs和outputs中的所有条目都是数值类型，它们可以转换为张量格式。 当数据采用张量格式后，可以通过在 2.1节中引入的那些张量函数来进一步操作。"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "outputs": [
    {
     "data": {
      "text/plain": "(tensor([[2., 1., 0.],\n         [1., 1., 0.],\n         [3., 0., 1.],\n         [2., 0., 1.]], dtype=torch.float64),\n tensor([127500, 124500, 123200, 123200]))"
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "X,y = torch.tensor(inputs.values),torch.tensor(outputs.values)\n",
    "X,y"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 2.2.4 练习\n",
    "\n",
    "创建包含更多行和列的原始数据集。\n",
    "1. 删除缺失值最多的列。\n",
    "2. 将预处理后的数据集转换为张量格式。"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   NumRooms  Alley   Prices\n",
      "0       NaN   Pave   127500\n",
      "1       1.0   Pave   124500\n",
      "2       3.0    NaN   123200\n",
      "3       NaN    NaN   123200\n"
     ]
    }
   ],
   "source": [
    "print(data)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "outputs": [
    {
     "data": {
      "text/plain": "NumRooms    2\n Alley      2\n Prices     4\ndtype: int64"
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.count(axis='index')"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "outputs": [
    {
     "data": {
      "text/plain": "   Alley   Prices\n0   Pave   127500\n1   Pave   124500\n2    NaN   123200\n3    NaN   123200",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Alley</th>\n      <th>Prices</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>Pave</td>\n      <td>127500</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>Pave</td>\n      <td>124500</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>NaN</td>\n      <td>123200</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>NaN</td>\n      <td>123200</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.drop(data.count(axis='index').idxmin(),axis=1)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ml",
   "language": "python",
   "name": "ml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}